Embodied Conversational Agents as a UI Paradigm: A Framework for Evaluation

Research on embodied conversational agent interfaces has produced widely divergent results. We suggest that this is due to insufficient consideration of key factors that influence the perception and effectiveness of agent-based interfaces. Thus, we propose a framework for the evaluation of conversational agent interfaces that can systematize the research. The framework emphasizes features of the agent, characteristics of the user, and the task the user is performing. We have conducted experiments within this framework. The first study manipulated the agent's appearance (lifelike versus iconic) and the nature of the user's task (carrying out procedures versus providing opinions). We found that the perception of the agent was strongly influenced by the task while features of the agent that we manipulated had little effect. The second study (in progress) manipulates the initiative of the agent (proactive versus reactive). Initial analysis of the data showed that the participants strongly preferred proactive agents while initiative of the agent had little effect on their task performance.